Size and Access Inference for Data-Parallel Programs (Draft Version)

Data-parallel programming languages have many desirable features, such as deterministic semantics and the ability to express large amounts of fine-grained parallelism. However, it is challenging to implement such languages efficiently on conventional MIMD multiprocessors, because these machines incur a high overhead for such small grain sizes. This paper presents analysis techniques for data-parallel program graphs that reduce these overheads in two ways: by stepping up the grain size, and by relaxing the synchronous nature of the computation without altering the program semantics. The algorithms work by partitioning the program graph into clusters of nodes such that all nodes in a cluster have the same iteration space size, and further refining these clusters into epochs based on generation and consumption patterns of data vectors. This converts the fine-grain parallelism in the original program to medium-grain loop parallelism, which is better suited to MIMD machines. We present performance results for data-parallel kernels analyzed by these techniques and converted to single-program multiple-data (SPMD) code running on an Encore Multimax.